{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Driving through the maze of traffic cones - Data Collection\n",
    "\n",
    "If you ran through the basic motion and collision avoidance notebooks, hopefully you're enjoying how easy it can be to make your Jetbot move around and avoid the obstacles like walls or chairs. In this notebook we will show how to teach JetBot *drive through the maze of traffic cones*!  \n",
    "\n",
    "Similar to collision avoidance notebook, we are going to attempt to solve the problem using deep learning classification model. The idea is to create a virtual \"safety bubble\" around the robot. Within this safety bubble, the robot is able to spin in a circle without hitting any objects. However, unlike collision avoidance example with only two classes (free or blocked) our classifier with try to determine which way to spin to stay on a path and by how much. The output of our model is a set of four probabilities:\n",
    "- **p(Left)** - a probability of turning left (spinning counterclockwise)\n",
    "- **p(right)** - a probability of turning right (spinning clockwise)\n",
    "- **p(blocked)** - a probability of the path being blocked\n",
    "- **p(free)** - a probability of no obstacles in front of the robot (so it is safe to move forward)\n",
    "\n",
    "This is how we collect the data:  \n",
    "\n",
    "First, we'll manually place the robot in scenarios where it's \"safety bubble\" is violated, and label these scenarios ``blocked``.  We save a snapshot of what the robot sees along with this label.\n",
    "\n",
    "Second, we'll manually place the robot in scenarios where it's safe to move forward a bit, and label these scenarios ``free``.  Likewise, we save a snapshot along with this label.\n",
    "\n",
    "Thrird, we'll manually place the robot in scenarios where spinning to the left (counterclockwise) would be the optimal move and label these scenarios ``left``. Likewise, we save a snapshot along with this label. Try to vary the angle of the desired rotation - place the robot in scenarios where this angle is larger or smaller.\n",
    "\n",
    "Finally, we'll manually place the robot in scenarios where turning right (clockwise) would be the optimal move and label these scenarios ``right``. Likewise, we save a snapshot along with this label. Try to vary the angle of the desired rotation - place the robot in scenarios where this angle is larger or smaller. \n",
    "\n",
    "Once we have 100-200 images and labels for each of four classes we have to options we can \n",
    "1. Upload this data to a GPU enabled machine where we'll *train* a neural network to predict the probabilities above  based off of the image it sees, OR\n",
    "2. Train model on JetBot's GPU  \n",
    "\n",
    "We'll use our trained model to implement a simple driving behavior in the end :)\n",
    "\n",
    "> IMPORTANT NOTE:  When JetBot spins in place, it actually spins about the center between the two wheels, not the center of the robot chassis itself.  This is an important detail to remember when you're trying to estimate whether the robot's safety bubble is violated or not. If in doubt it's better to lean on the cautious side (a big safety bubble).  We want to make sure JetBot doesn't enter a scenario that it couldn't get out of by turning in place."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display live camera feed\n",
    "\n",
    "First, let's initialize and display our camera like we did in the *teleoperation* notebook.  \n",
    "\n",
    "> Our neural network takes a 224x224 pixel image as input.  We'll set our camera to that size to minimize the filesize of our dataset (we've tested that it works for this task).\n",
    "> In some scenarios it may be better to collect data in a larger image size and downscale to the desired size later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7600a43eea6b47f491a597fd6a720011",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Image(value=b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00\\x00\\x01\\x00\\x01\\x00\\x00\\xff\\xdb\\x00C\\x00\\x02\\x01\\x0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import traitlets\n",
    "import ipywidgets.widgets as widgets\n",
    "from IPython.display import display\n",
    "from jetbot import Camera, bgr8_to_jpeg\n",
    "\n",
    "camera = Camera.instance(width=224, height=224)\n",
    "\n",
    "image = widgets.Image(format='jpeg', width=224, height=224)  # this width and height doesn't necessarily have to match the camera\n",
    "\n",
    "camera_link = traitlets.dlink((camera, 'value'), (image, 'value'), transform=bgr8_to_jpeg)\n",
    "\n",
    "display(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the cell above JetBot should display the camera feed. \n",
    "\n",
    "If you observe a substantial time lag, there are a couple of things you can try:\n",
    "\n",
    "1. Make sure you use the listed battery pack and it is charged\n",
    "\n",
    "2. Check if Jetson Nano is in MAXN or 5W mode. You can determine this by calling the following from a terminal:\n",
    "\n",
    "``nvpmodel -q``\n",
    "\n",
    "3. Move your JetBot closer to WiFi router\n",
    "\n",
    "4. Restart nvargus-daemon (details are [here](https://github.com/NVIDIA-AI-IOT/jetbot/issues/47)):\n",
    "\n",
    "``sudo systemctl restart nvargus-daemon``\n",
    "\n",
    "5. Shutdown all kernels (*Kernel -> Shudown All Kernels*) and restart the notebook Move your JetBot closer to WiFi router\n",
    "\n",
    "6. Stop and disable [rsyslog](https://github.com/NVIDIA-AI-IOT/jetbot/issues/63):\n",
    "\n",
    "``sudo service rsyslog stop``\n",
    "\n",
    "``sudo systemctl disable rsyslog``\n",
    "\n",
    "If nothing helps, you should still be able to collect the data - you just cannot rely on the display with the camera feed which is ammoying but not critical.)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a few directories where we'll store all our data.  We'll create a folder ``dataset_cones`` that will contain four sub-folders ``free``, ``left``, ``right`` and ``blocked`` where we'll place the images for each scenario."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Directories not created becasue they already exist\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "right_dir = 'dataset_cones/right'\n",
    "left_dir = 'dataset_cones/left'\n",
    "free_dir = 'dataset_cones/free'\n",
    "blocked_dir = 'dataset_cones/blocked'\n",
    "\n",
    "# we have this \"try/except\" statement because these next functions can throw an error if the directories exist already\n",
    "try:\n",
    "    os.makedirs(free_dir)\n",
    "    os.makedirs(right_dir)\n",
    "    os.makedirs(left_dir)\n",
    "    os.makedirs(blocked_dir)\n",
    "except FileExistsError:\n",
    "    print('Directories not created becasue they already exist')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you refresh the Jupyter file browser on the left, you should now see those directories appear.  Next, let's create and display some buttons that we'll use to save snapshots\n",
    "for each class label.  We'll also add some text boxes that will display how many images of each category that we've collected so far. This is useful because we want to make\n",
    "sure we collect about the same number of images for each class (``free``, ``left``, ``right`` or ``blocked``.)  It also helps to know how many images we've collected overall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "button_layout = widgets.Layout(width='128px', height='64px')\n",
    "free_button = widgets.Button(description='add free',   button_style='success', layout=button_layout)\n",
    "right_button = widgets.Button(description='add right', button_style='info', layout=button_layout)\n",
    "left_button = widgets.Button(description='add left',   button_style='warning', layout=button_layout)\n",
    "blocked_button = widgets.Button(description='add blocked',   button_style='danger', layout=button_layout)\n",
    "\n",
    "free_count = widgets.IntText(layout=button_layout,  value=len(os.listdir(free_dir)))\n",
    "right_count = widgets.IntText(layout=button_layout, value=len(os.listdir(right_dir)))\n",
    "left_count = widgets.IntText(layout=button_layout,  value=len(os.listdir(left_dir)))\n",
    "blocked_count = widgets.IntText(layout=button_layout,  value=len(os.listdir(blocked_dir)))\n",
    "\n",
    "# display(widgets.HBox([right_count, right_button]))\n",
    "# display(widgets.HBox([free_count, free_button]))\n",
    "# display(widgets.HBox([left_count, left_button]))\n",
    "# display(widgets.HBox([blocked_count, blocked_button]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right now, these buttons wont do anything.  We have to attach functions to save images for each category to the buttons' ``on_click`` event.  We'll save the value\n",
    "of the ``Image`` widget (rather than the camera), because it's already in compressed JPEG format!\n",
    "\n",
    "To make sure we don't repeat any file names (even across different machines!) we'll use the ``uuid`` package in python, which defines the ``uuid1`` method to generate\n",
    "a unique identifier.  This unique identifier is generated from information like the current time and the machine address."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from uuid import uuid1\n",
    "\n",
    "def save_snapshot(directory):\n",
    "    image_path = os.path.join(directory, str(uuid1()) + '.jpg')\n",
    "    with open(image_path, 'wb') as f:\n",
    "        f.write(image.value)\n",
    "\n",
    "def save_free():\n",
    "    global free_dir, free_count\n",
    "    save_snapshot(free_dir)\n",
    "    free_count.value = len(os.listdir(free_dir))\n",
    "    \n",
    "def save_right():\n",
    "    global right_dir, right_count\n",
    "    save_snapshot(right_dir)\n",
    "    right_count.value = len(os.listdir(right_dir))\n",
    "\n",
    "def save_left():\n",
    "    global left_dir, left_count\n",
    "    save_snapshot(left_dir)\n",
    "    left_count.value = len(os.listdir(left_dir))\n",
    "    \n",
    "def save_blocked():\n",
    "    global blocked_dir, blocked_count\n",
    "    save_snapshot(blocked_dir)\n",
    "    blocked_count.value = len(os.listdir(blocked_dir))\n",
    "\n",
    "# attach the callbacks, we use a 'lambda' function to ignore the\n",
    "# parameter that the on_click event would provide to our function\n",
    "# because we don't need it.\n",
    "free_button.on_click(lambda x: save_free())\n",
    "right_button.on_click(lambda x: save_right())\n",
    "left_button.on_click(lambda x: save_left())\n",
    "blocked_button.on_click(lambda x: save_blocked())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Now the buttons above should save images to the ``free``,  ``left`` and ``right`` directories.  You can use the Jupyter Lab file browser to view these files!\n",
    "\n",
    "Now go ahead and collect some data \n",
    "\n",
    "1. Place the robot in a scenario where it's supposed to turn right and press ``add right``\n",
    "2. Place the robot in a scenario where it's supposed to turn left and press ``add left``\n",
    "3. Place the robot in a scenario where it's free and press ``add free``\n",
    "3. Place the robot in a scenario where it's blocked and press ``add blocked``\n",
    "5. Repeat 1, 2, 3, 4\n",
    "\n",
    "> REMINDER: You can move the widgets to new windows by right clicking the cell and clicking ``Create New View for Output``.  Or, you can just re-display them\n",
    "> together as we will below\n",
    "\n",
    "Here are some tips for labeling data\n",
    "\n",
    "1. Try different orientations (e.g. sharp right vs slight right, closer to the cone or further away from it, etc.) \n",
    "2. Try different lighting\n",
    "3. Try different textured floors / objects;  patterned, smooth, glass, etc.\n",
    "\n",
    "Ultimately, the more data we have of scenarios the robot will encounter in the real world, the better our collision avoidance behavior will be.  It's important\n",
    "to get *varied* data (as described by the above tips) and not just a lot of data, but you'll probably need at least 100 images of each class (that's not a science, just a helpful tip here).  But don't worry, it goes pretty fast once you get going :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7600a43eea6b47f491a597fd6a720011",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Image(value=b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00\\x00\\x01\\x00\\x01\\x00\\x00\\xff\\xdb\\x00C\\x00\\x02\\x01\\x0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9bcf1d940f474ca683513905deb6c8fa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Button(button_style='warning', description='add left', layout=Layout(height='64px', width='128p…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51c210c5c4384ee68d7f117b6bbda1f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntText(value=180, layout=Layout(height='64px', width='128px')), VBox(children=(IntText(value=1…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(image)\n",
    "# display(widgets.HBox([right_count, right_button]))\n",
    "# display(widgets.HBox([free_count, free_button]))\n",
    "# display(widgets.HBox([left_count, left_button]))\n",
    "# display(widgets.HBox([blocked_count, blocked_button]))\n",
    "\n",
    "# display buttons\n",
    "middle_box = widgets.VBox([free_button, blocked_button])\n",
    "controls_box = widgets.HBox([left_button, middle_box, right_button])\n",
    "display(controls_box)\n",
    "\n",
    "# display counts\n",
    "middle_box_count = widgets.VBox([free_count, blocked_count])\n",
    "controls_box_count = widgets.HBox([left_count, middle_box_count, right_count])\n",
    "display(controls_box_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next\n",
    "\n",
    "Once you've collected enough data, we'll need to copy that data to our GPU desktop or cloud machine for training.  First, we can call the following *terminal* command to compress\n",
    "our dataset folder into a single *zip* file.\n",
    "\n",
    "> The ! prefix indicates that we want to run the cell as a *shell* (or *terminal*) command.\n",
    "\n",
    "> The -r flag in the zip command below indicates *recursive* so that we include all nested files, the -q flag indicates *quiet* so that the zip command doesn't print any output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "!zip -r -q dataset_cones.zip dataset_cones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see a file named ``dataset_cones.zip`` in the Jupyter Lab file browser.  You should download the zip file using the Jupyter Lab file browser by right clicking and selecting ``Download``.\n",
    "\n",
    "> NOTE: If Chrome blocks download from the notebook you can use Internet Explorer for downloading (i.e. connect to JetBot from Internet Explorer and download the data) \n",
    "\n",
    "Next, we'll need to upload this data to our GPU desktop or cloud machine (we refer to this as the *host*) to train the collision avoidance neural network.  We'll assume that you've set up your training\n",
    "machine as described in the JetBot WiKi.  If you have, you can navigate to ``http://<host_ip_address>:8888`` to open up the Jupyter Lab environment running on the host.  The notebook you'll need to open there is called ``collision_avoidance/train_model_cones.ipynb``.\n",
    "\n",
    "So head on over to your training machine and follow the instructions there!  Once your model is trained, we'll return to the robot Jupyter Lab enivornment to use the model for a live demo!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
